Skip to content

spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode - #541

Draft
lu-zero wants to merge 2 commits into
mudler:mainfrom
lu-zero:row/BACKEND-TENSTORRENT-TRACE-RUNNER
Draft

spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode#541
lu-zero wants to merge 2 commits into
mudler:mainfrom
lu-zero:row/BACKEND-TENSTORRENT-TRACE-RUNNER

Conversation

@lu-zero

@lu-zero lu-zero commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What

BACKEND-TENSTORRENT-TRACE-RUNNER (child of BACKEND-TENSTORRENT). A read-only feasibility spike (POL-SPIKE-FIRST, POL-NO-CEILING) deciding whether to wire the landed graph-capture foundation (#354 / 59568772) into a capturable decode region. No code shipped — decision record only. The spike actually attempted the capture on-card rather than guessing.

Findings

Q1 — host-free region at T=1? No. Static trace: RmsNormKernel (rows=1 < 32) and PreferDeviceRope (T·H = 16 < 64) both route to host every layer. Bonus: the shared decode-graph framework Qwen3DenseDecodeGraph (qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) is disabled for TT because the platform doesn't override support_static_graph_mode() (base false; only CUDA returns true).

Q2 — all-device-at-T=1 tok/s cost? Measured on real Blackhole P150, Qwen3-0.6B, local-only VT_TT_FORCE_DEVICE override (reverted, unshipped):

config warm tok/s
hybrid (current default) 12.5
all-device (forced) 10.7

Reproduces handoff §6 precisely (~12.3→10.5; 12.5→10.7 here; same ~14% / ~1.8 tok/s regression).

Q2b — capture attempted on-card: ABORTS. With both support_static_graph_mode()true (enables Qwen3DenseDecodeGraph, which wires BeginCapture/ForwardLayers/EndCaptureGraph) AND VT_TT_FORCE_DEVICE set, capture still fails:

TT_FATAL: Reads are not supported during trace capture
  backtrace: ttnn::Tensor::to_vector<float>
[Qwen3DenseDecodeGraph] 0 total replays across 1 captured size(s)

A device→host readback fires inside the captured ForwardLayers region. ttnn begin_trace_capture prohibits any host read during capture; forcing the two hybrid thresholds all-device is insufficient because other ops (embedding / paged-attention / logits) still do to_vector.

Decision

NO-GO for pure T=1 decode capture — measured, not assumed. Decode capture cannot run at all on the current forward without first eliminating every to_vector readback in the captured region, which is a redesign of the TT forward's host-staging model, not a threshold tweak. The earlier "implausible to recover 1.8 tok/s" reasoning is superseded: capture aborts before perf even enters the picture.

Open follow-on (separate row): prefill / multi-token chunk capture. At T≥32 the thresholds already go all-device; whether the prefill region is also free of to_vector readbacks is the open question the next row must answer first (same ttnn host-read prohibition applies).

How the other backends do it

CUDA is the ONLY SupportsGraphCapture()==true backend (cuda_backend.cu:184-240: contract = async/no host sync/no malloc in region, fixed pointers). Metal and Vulkan are false with the eventual mapping named but unimplemented. Model-side capture sites capture ForwardLayers device-resident, one eager step warms the pool first.

Files

  • NEW .agents/specs/tenstorrent-trace-runner.md (spec + measured ## Outcome + decision)
  • NEW BACKEND-TENSTORRENT-TRACE-RUNNER row in .agents/backend-matrix.md (SPIKE, decision record complete)
  • NEW claim in .agents/coordination.md
  • scripts/check-agent-record.py BACKEND 81→82
  • No src/ change. Both local overrides (support_static_graph_mode, VT_TT_FORCE_DEVICE) were reverted.

@lu-zero
lu-zero force-pushed the row/BACKEND-TENSTORRENT-TRACE-RUNNER branch from 690b49e to f0bdc2d Compare August 12, 2026 22:16
localai-bot added a commit that referenced this pull request Aug 12, 2026
…_topk -- remove the literal bounds (#505)

Lands the #505 fix: DsaTopkKernel sized `bool chosen[512]` and `int64_t
picked[64]` by literal while index_topk is 512 (V4-Flash) / 1024 (V4-Pro).
MEASURED on dgx.casa (GB10, sm_121a): the pre-fix kernel at Flash's own width
takes `cudaStreamDestroy: an illegal memory access` and SIGABRTs; after the
two-pass threshold rewrite the suite is 23/23 with 83913/83913 assertions and
0 skipped, re-verified from the merged tree with CUTLASS + FlashAttention-2
hard-verified in that run's own configure log.

The fix removes the bounds rather than asserting them, so there is no
configurable limit left to outgrow, and it drops the O(topk^2) emit sort.

FRESH REVIEW: PASS. An independent reviewer reproduced both the defect and the
fix on real sm_121a hardware, fuzzed 3,000,081 shapes across three independent
implementations (host reference, kernel transcription, and its own O(n^2)
rank-count oracle) with ZERO divergence, and ran a 12-row device mutation table.
It found no blocking issue and 6 non-blocking findings, addressed in a follow-up.
Notably it confirmed the tie-heavy case is uniquely load-bearing: it is the only
case that catches a tie-break inversion or a value-only threshold.

CI: windows-msvc-cpu and windows-msvc-vulkan are red at the repo-wide BASELINE,
not from this change -- the same two jobs fail on #539, #541 and on #511 which
already merged, and this PR's own log shows every target building with no
`error C####`, failing instead in the release-packaging PowerShell step.

Merged on direct developer instruction after the requested review pass.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot added a commit that referenced this pull request Aug 13, 2026
…opk, and correct two guard-comment overclaims (#552)

Lands the six findings from the fresh review of #505, which returned PASS after
failing to find any input where the two-pass threshold selection diverges from
`DsaTopkSelect` -- 3,000,081 fuzzed shapes across three independent
implementations, zero divergence.

CODE: `DsaTopkLaunch` now checks `cudaGetLastError()` after the launch and refuses
non-positive `topk` to match the host reference, which previously asserted while
the device arm silently returned an empty vector.

TESTS: two new device cases. One drives the window clamps -- under-run, over-run,
both at once, and an in-range control -- which nothing exercised before. One
asserts BOTH arms refuse topk=0 and topk=-1.

PROSE: the `w < topk` guard comment claimed to defend against a NaN row writing
past its own row. That cannot occur: pass 2's predicate is satisfied by exactly
rank(th) <= topk elements and a NaN never satisfies it either. Corrected to
belt-and-braces, bounded by construction. "Strictly cheaper" is now scoped to a
comparison with the old kernel, not fitness for the real geometry.

REFUTED IN FLIGHT: finding 4 held that the launch check would make the next fault
attributable to this launch. The arm built to demonstrate it disproved it -- the
pre-fix kernel WITH the check still reports `cudaStreamDestroy: an illegal memory
access`, because a stack-overflow fault is asynchronous while `cudaGetLastError()`
covers launch-configuration errors. The check is kept and its comment now says
what it actually does; claiming otherwise would have repeated finding 1's defect
inside the change that exists to correct one.

DEVICE ARMS on dgx.casa (GB10 sm_121a), each a fresh nvcc rebuild from a pristine
kernel with the binary mtime verified to advance, scoped with `-tc=` never `-ts=`,
under `flock $HOME/gpu.lock`: baseline 6/6 SUCCESS; no_topk_guard FAILURE;
no_ws_clamp exit=134 CRASHED; no_we_clamp 1052 failed assertions; no_launch_check
SUCCESS (unobservable by construction); prefix_with_check exit=134; restored full
suite 25/25 with 90062 assertions, 0 skipped, SUCCESS. Both new cases have teeth
against the mutation each was written for.

CPU 25/25 and 13/13, preflight fully clean including the contention-sensitive
floor harness. Windows MSVC red is the repo-wide baseline (same two jobs fail on
#539, #541 and on #511 which already merged; no `error C####`, the failure is the
release-packaging PowerShell step).

Merged on direct developer instruction.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
BACKEND-TENSTORRENT-TRACE-RUNNER. Replaces the prior "no-go" guess with a
measured result from actually attempting the capture on-card.

The spike ran the real experiment: local-only flips of both
support_static_graph_mode() (enables Qwen3DenseDecodeGraph) AND
VT_TT_FORCE_DEVICE (RoPE+residual all-device), then Qwen3-0.6B cli smoke.
Both overrides reverted; no code shipped.

Result: capture ABORTS. ttnn raises
  TT_FATAL: Reads are not supported during trace capture
with a backtrace through ttnn::Tensor::to_vector<float> -- a device->host
readback fires inside the captured ForwardLayers region. 0 replays. The
T=1 forward does to_vector readbacks (embedding/PA/logits) that ttnn trace
prohibits; forcing the two hybrid thresholds all-device is insufficient.

This is strictly stronger than the earlier tok/s-recovery guess: decode
capture cannot run AT ALL on the current forward without eliminating every
to_vector readback in the captured region (a host-free ForwardLayers
redesign, not a threshold tweak).

Q1 (no host-free region at T=1) and Q2 (all-device costs 12.5->10.7 tok/s,
reproducing handoff §6) stand. Decision: NO-GO for pure T=1 decode, measured.
Next: prefill capture (separate row) must first audit its to_vector readbacks.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Maki:glm-5.2 [Maki]
BACKEND-TENSTORRENT-HOST-FREE-FORWARD (child of BACKEND-TENSTORRENT). The
trace-runner spike measured that decode mesh-trace capture ABORTS
(TT_FATAL: Reads are not supported during trace capture, backtrace through
to_vector<float>) because the per-layer T=1 forward does device->host
readbacks that ttnn trace prohibits. Capture wiring works (it engaged and
tried); the forward is the blocker.

This commit is a PLAN only (no code). It decomposes the host-free forward
into four independent rows, grounded in a full readback map of the current
TT ops:

  R1 RmsNorm + RoPE all-device at T=1 (threshold flip; numerics already
     proven by BACKEND-TENSTORRENT-RESIDUAL-GOLDEN; eager cost 12.5->10.7
     tok/s measured, recovered by capture).
  R2 QkvSplit + ReshapeAndCache device-resident (bit-exact memcpy/stride
     ops; reuse the landed paged-KV-shadow path).
  R3 PagedAttention decode with device-resident metadata (the scope risk --
     device SDPA decode exists, but block_table/seq_lens/query_start_loc
     are read host-side around it).
  R4 flip support_static_graph_mode + wire capture + measure replay tok/s
     (integration; the payoff that justifies R1-R3: replay warm tok/s
     >= 12.5).

R1-R3 parallel-claimable; R4 integrates. The R1 eager regression (12.5->
10.7) is acceptable ONLY because R4 recovers it; R1 must gate on
support_static_graph_mode() so non-capture runs keep the 12.5 baseline.

The trace-runner row's "next" now points here instead of to a vague
prefill-capture note: the host-free forward is the real unblock.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Maki:glm-5.2 [Maki]
@lu-zero
lu-zero force-pushed the row/BACKEND-TENSTORRENT-TRACE-RUNNER branch from 13f35a9 to ec41961 Compare August 13, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant